Network interconnect as a switch

ABSTRACT

An interconnect as a switch module (“ICAS” module) comprising n port groups, each port group comprising n-1 interfaces, and an interconnecting network implementing a full mesh topology where each port group comprising a plurality of interfaces each connects an interface of one of the other port groups, respectively. The ICAS module may be optically or electrically implemented. According to the embodiments, the ICAS module may be used to construct a stackable switching device and a multi-unit switching device, to replace a data center fabric switch, and to build a new, high-efficient, and cost-effective data center.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional application of U.S. application Ser.No. 17/349,628, filed on Jun. 16, 2021, which is a divisionalapplication of U.S. application Ser. No. 16/921,264, filed on Jul. 6,2020, which is the divisional application of U.S. application Ser. No.16/257,653 filed on Jan. 25, 2019, which is a Continuation-in-Partapplication of U.S. application Ser. No. 15/888,516 filed on Feb. 5,2018, all of which are incorporated herein by reference in theirentireties, including any figures, tables, and drawings.

FIELD OF INVENTION

The present invention relates to computer network. In particular, thepresent invention relates to interconnecting structure of ICAS module,stackable switching device, multi-unit chassis switching device, networkpod, fanout cable transpose rack and datacenter network.

DISCUSSION OF THE RELATED ART

As a result of the recent rapid growth in application needs—in both sizeand complexity—today's network infrastructure is scaling and evolving ata high rate. The data traffic that flows from a data center to theInternet—i.e., “machine-to-user” traffic—is large, and ever increasing,as more people get connected, and as new products and services arecreated. However, machine-to-user traffic is merely “the tip of theiceberg,” when one considers the data traffic within the datacenter—i.e., “machine-to-machine” traffic—necessary to generate themachine-to-user data traffic. Generally, machine-to-machine data trafficis several orders of magnitude larger than machine-to-user data traffic.

The back-end service tiers and applications are distributed andlogically interconnected within a data center. To service each user whouses an application program (“app”) or a website, these back-end servicetiers and applications rely on extensive real-time “cooperation” witheach other to deliver the user's expected customized fast and seamlessexperience at the front end. To keep up with the demand, even though theinternal applications are constantly being optimized to improveefficiency, the corresponding machine-to-machine traffic grows at aneven faster rate than their continual optimization (e.g., at the currenttime, machine-to-machine data traffic is growing roughly faster thandoubling every year).

To be able to move fast and to support rapid growth are goals that areat the core of data center infrastructure design philosophy. Inaddition, the network infrastructure within a data center (“data centernetwork”) must be simple enough as to be managed by a small, highlyefficient team of engineers. It is desired that the data center networkevolves in the direction that makes deploying and operating the networkeasier and faster over time, despite scale and exponential growth.

Some of these applications needs relate to the increasing use of dataanalytic tools (“big data”) and artificial intelligence (“AI”), forexample. As discussed above, big data and AI have become verysignificant distributed applications. Servicing these applicationsrequire handling large amounts of data (e.g., petabytes), using greatcomputation power (e.g., petaflops), and achieving very low latency(e.g., responses that become available within 100 ns). Simultaneouslyproviding more powerful processors (“scaling-up”) and exploiting greaterparallel processing (“scaling-out”) have been the preferred approach toachieve performance. Unlike scientific computation, however, big-dataand AI applications are delivered in the form of services to largenumbers of users across the world. Thus, like web servers for webservices, clusters of servers dedicated to big data and AI applicationshave become significant parts of the data center network.

At the current time, data center networks have largely transitioned fromlayer-2 to all layer-3 (e.g., running Border Gateway Protocol (BGP) andEqual-cost Multi-Path (ECMP) protocols). A large-scale data center todayis typically operating at tens of petabits-per-second scale (petascale)and expects growth into the hundreds of petabits-per-second scale in thenear future. The cost of provisioning such data center ranges fromUS$300 million to US$1.5 billion.

Let us define several terms in Table 1, before proceeding with thedescription of this patent.

TABLE 1 Terminology Description media The media is a concept of aphysical entity. It can be optical or electronic. interface An interfaceis a concept of a physical entity. It contains a transmission media anda reception media. The media can be optical or electrical. An interfacecan associate with a MAC (Medium Access Control) entity or severalinterfaces can associate with a MAC entity. port A port is a concept ofa container entity. It includes a set of interfaces. The number of theinterfaces depends on the technology. For example, a 40 G QSFP Ethernetport consists of 4 interfaces (a total of 8 fibers). The 40 G QSFP portassociates with a MAC entity. The 10/40 G optical technology isimplemented by reconfiguring the 40 G QSFP Ethernet to 4 independent 10G interfaces each associates with a MAC entity. As such, each interfacecan provide connectivity and operate like a port. port group A portgroup is a concept of a container entity; it includes a set of ports;the number of ports depends on application. Each interface in a portgroup is configured to associate with a MAC presumably. In order to meetthe needs of description, this patent introduces and defines the term of“port group”. connection A connection consists of media and twointerfaces communicating through the media. link A link is a concept ofa container entity. It includes a set of connections. The number of theconnections depends on the technology. For example, two 40 G QSFPconnectors docked together to form one link which contains 4connections; two 10/40 G QSFP connectors docked together to form 4 linkseach contains 1 connection. downlink Refers to the link that connectstoward the hosts. uplink Refers to the link that connects toward thecore of the network. intralink Refers to the link that providesconnectivity inside a pod. interlink Refers to the link that providesconnectivity between two pods.

A review of our current state-of-the-art data center infrastructure isinstructive. In the following context, data or traffic aggregationrefers to multiplexing of communication frames or packets. Aggregationmodel and disaggregation model refer to topologies of communicationnetworks. The concept of data aggregation is orthogonal to the conceptof an aggregation or disaggregation model. Therefore, a disaggregationmodel can support data aggregation, as discussed below. A basic conceptin data communication is that communication channels can be error prone.Transmission over such communication channels at higher data rates andover large distances requires complex and costly transceivers.Consequently, channel encoding, error detection and correction, andcommunication protocols are many techniques to ensure data istransmitted over long distances with accuracy. In the past, as datatransmission was expensive, data aggregation (e.g., multiplexing fromdifferent data streams and multiplexing data from multiple topologicalsources) and data compression ensure even higher utilization of thecommunication channels and efficient management of the communicationcost. This is the origin of the aggregation (i.e., in both data andtopology) paradigm. This paradigm dominates the networking industry fordecades. Such aggregation is widely used in wide area networks (WANs),where transmission cost dominates over other network costs. Today'shardware architecture for data switching is also based on aggregation,i.e., each port is connected and aggregated from all other port. Intoday's communication networks, data is typically aggregated beforetransmitting on to “uplink” to connect to external network (e.g., theInternet), which tends to be the most expensive port of the dataswitching equipment. Due to both advances in semiconductor,fiber-optical, and interconnect technologies and economy of scale,network costs have reduced significantly. The aggregation model is notnecessarily the only—or the most suitable—solution in a data center. Intoday's data center networks, where machine-to-machine traffic(“east-west traffic”) dominates most of the bandwidth, being severalorders of magnitude than the machine-to-user bandwidth, multipathtopology and routing (ECMP) are deployed so that the combined networkbandwidth is large. However, traffic is still aggregated from allincoming port on to each outgoing port. Nonetheless, the multipathtopology signifies a disaggregation model. The detailed descriptionbelow places a structure and quantification onto the multipath topologyand discloses a disaggregation model, referred to herein as“interconnect as a Switch” (“ICAS”), which is significantly differentfrom the more traditional aggregation model for data centers.

Typically, in an enterprise or intranet environment, communicationpatterns are relatively predictable with a modest number of data sourcesand data destinations. These data sources and data destinations aretypically connected by a relatively small number of designated paths(“primary paths”), with some number of back-up or “secondary paths,”which are provided primarily for fault tolerance. In such anenvironment, the routing protocols of the enterprise network areoptimized to select a shortest single path between eachsource-destination pair in the absence of a failure.

Distributed computing frameworks (e.g., MapReduce, Hadoop and Dryad) andweb services (e.g., web search, ecommerce, social networking, dataanalytics, artificial intelligence and scientific computing) bring a newparadigm of computing that requires both interconnections between adiverse range of hosts and significant aggregate bandwidths. Due to thescarcity of ports even in the high-end commercial switches, a commonhierarchical network topology that has evolved is a fat tree withhigher-speed ports and increasing aggregate bandwidths, as one moves upthe hierarchy (i.e., towards the roots). The data center network, whichrequires substantial intra-cluster bandwidths, represents a departurefrom the earlier hierarchical network topology. In the multi-rootedtree, the shortest single-path routing protocol can significantlyunderutilize the available bandwidths. The ECMP is an improvement thatstatically stripes flows across available paths using flow hashingtechniques. ECMP is standardized in the IEEE 802.1Q Standard. ECMPallows “next-hop packet forwarding” to a single destination to occurover multiple “best paths,” as symmetric insuring flows on deterministicpaths. Equal cost multi-path routing can be used in conjunction withmost routing protocols, because it is a per-hop decision limited to asingle router. It can substantially increase bandwidth by load-balancingtraffic over multiple paths. When a data packet of a data stream arrivesat the switch, and multiple candidate paths are available for forwardingthe data packet to its destination, selected fields of the data packet'sheaders are hashed to select one of the paths. In this manner, the flowsare spread across multiple paths, with the data packets of each flowtaking the same path, so that the arrival order of the data packets atthe destination is maintained.

Note that ECMP performance intrinsically depends on both flow size andthe number of flows arriving at a host. A hash-based forwarding schemeperforms well in uniform traffic, with the hosts in the networkcommunicating all-to-all with each other simultaneously, or in whichindividual flow last only a few round-trip delay times (“RTTs”).Non-uniform communication patterns, especially those involving transfersof large blocks of data, do not perform well under ECMP without carefulscheduling of flows to avoid network bottlenecks.

In the detailed description below, the terms “fabric switch” and “spineswitch” are used interchangeably. When both terms appear in a network, afabric switch refers to a device in a network layer which is used formultipath networking among TOR devices, while a spine switch refers to adevice in a higher network layer which is used for multipath networkingamong pods.

A fat tree network suffers from three types of drawbacks—i.e., 1)congestion due to hash collision, 2) congestion due to an aggregationmodel, and 3) congestion due to a blocking condition. These congestionsare further examined in the following.

First, under ECMP, two or more large, long-lived flows can hash to thesame path (“hash collision”), resulting in congestion, as illustrated inFIG. 1 a. FIG. 1 a shows four fabric switches 10-0 to 10-3interconnecting five TOR switches 11-0 to 11-4. As shown in FIG. 1 a,each TOR switch has four ports each communicating with a port of one ofthe fabric switches 10-0 to 10-3, respectively. Each fabric switch hasfive ports each communicating with a port of one of TOR switches 11-0 to11-4, respectively. In FIG. 1 a, two flows designating TOR switch 11-0are sourced from TOR switches 11-1 and 11-2. However, by chance, eachflow is hashed to a path that goes through fabric switch 10-0, whichcauses congestion at designating port 101 of fabric switch 10-0. (Ofcourse, the congestion problem could have been avoided if one of theflows is hashed to a path that goes through fabric switch 10-1 forinstance). Furthermore, the static mapping of the flows to paths byhashing does not consider either current network utilization or thesizes of the flows, so that the resulting collision overwhelms switchbuffers, degrade overall switch utilization, and increases transmissionlatency.

Second, in a fat tree network, the total bandwidth of the aggregatedtraffic may exceed the bandwidth of all the downlinks of all the fabricswitches facing the same TOR switch, resulting in aggregationcongestion, as shown in FIG. 1 b. Such aggregation congestion is acommon problem in the aggregation model of today's switching network,and requires detailed rate limiting to avoid congestion. In FIG. 1 b,the traffic through the fabric switches 12-0 to 12-3 facing the TORswitch 13-0 is sourced from the TOR switches 13-1 to 13-4, but theaggregate traffic from the source (one source each from TOR switches13-1 to 13-3 and two sources from TOR switch 13-4) exceeds the combinedbandwidth of all the downlinks of all the fabric switches 12-0 to 12-3facing the TOR switch 13-0. More specifically, traffic is spread outevenly over fabric switch 12-1 to 12-3 without congestion; additionaltraffic from TOR switch 13-4 exceeds the downlink bandwidth of port 121of fabric switch 12-0 and thus causes congestion.

Third, there is a blocking condition called the “strict-sense blockingcondition,” which is applicable to statistically multiplexed flow-basednetworks (e.g., a TCP/IP network). The blocking condition results frominsufficient path diversity (or an inability to explore path diversityin the network) when the number and the size of the flows becomesufficiently large. FIG. 1 c illustrates the blocking condition in a fattree network. As shown in FIG. 1 c, the blocking condition occurs, forexample, when paths from fabric switches 14-0 and 14-1 to TOR switch15-0 are busy and paths from fabric switches 14-2 and 14-3 to TOR switch15-3 are busy, and a flow which requires a path through TOR switch 15-0arrives at TOR switch 15-3. An extra flow between TOR switch 15-0 and15-3 can take one of 4 possible paths. Say it takes the path from TORswitch 15-3 to fabric switch 14-1 and then from fabric switch 14-1 toTOR switch 15-0. However, the path from fabric switch 14-1 to TOR switch15-0 is busy already. Overall, multiplexing the blocked flow on to theexisting flows results in increased congestion, latency and/or packetloss.

At the same time as the demand on the data center network grows, therate of growth in CMOS circuit density (“Moore's law”) and the I/Ocircuit data rate appear to have slowed. The cost of lithographic andheat density will ultimately limit how many transistors can be packedinto a single silicon package. That is to say, an ultra large storage orcomputing system is bound to be achieved through multiple chips. It isunlikely that an ultra large system will be integrated on a single chipwith ultra-high integration density as in the past. The question thatarises here is how to build an ultra large bandwidth interconnectionbetween the chips. It is instructive to learn that a switching chipsoldered on printed circuit board (PCB) employs high-speed serialdifferential I/O circuit to transmit and receive data to/fromtransceiver module. A transceiver module interconnects to a transceivermodule on a different system to accomplish network communications. Anoptical transceiver performs the electrical-to-optical andoptical-to-electrical conversion. An electrical transceiver performscomplex electrical modulation and demodulation conversion. The primaryobstacle that hinders high-speed operation on PCB is thefrequency-dependent losses of the copper-based interconnection due toskin effects, dielectric losses, channel reflections, and crosstalk.Copper-based interconnection faces the challenge of bandwidth limit asthe data rate exceeds several tens of gigabit per second (Gb/s). Tosatisfy demands for bigger data bandwidth high-radix switch siliconintegrates hundreds of differential I/O circuits. For example, BroadcomTrident-II chip and Barefoot Network Tofino chip integrate 2×128 and2×260 differential I/O circuits for 10 Gb/s transmit and receiverespectively. To optimize system level port density, heat dissipationand bandwidth the I/O circuits and interfaces are gathered in groups andstandardized in specifications on electrical and optical properties. ForSFP+, each port has a pair of TX and RX serial differential interfacesat 10 Gb/s data rate. For QSFP, each port has four pairs of TX and RXserial differential interfaces at 10 Gb/s data rate each for a total of40 Gb/s or 4×10 Gb/s data rate. For QSFP28, each port has four pairs ofTX and RX serial differential interfaces at 25 Gb/s data rate each for atotal of 100 Gb/s or 4×25 Gb/s data rate. For QSFP-DD, each port haseight pairs of TX and RX serial differential interfaces with a data rateof 50 Gb/s data rate each for a total of 400 Gb/s or 8×50 Gb/s datarate. State of the art data centers and switch silicon employ 4 or 8interfaces (TX, RX) at 10 Gb/s or 25 Gb/s or 50 Gb/s per port as designconsiderations. These groupings are not necessarily unique. MTP/MPO asan optical interconnect standard defines up to 48 interfaces per portwhere each interface contains a pair of optical fibers one for transmitand one for receive. However the electrical and optical specificationsof transceiver with up to 48 interfaces per module are yet to come. Thedefinition of “port group” in this patent disclosure is extended toinclude more interfaces crossing multiple ports (e.g., 8 interfaces from2 QSFP's; 32 interfaces from 8 QSFP's, etc.). A person experienced inthe art can understand that this invention is applicable to otherinterconnect standards where multiple various number of interfaces otherthan 4 be grouped together in the future.

These limitations affect data center networks by, for example,increasing power consumption, slowing of performance increase, andincreasing procurement cycle. These developments exacerbate the powerneeds for the equipment, as well as their cooling, facility space, thecost of hardware, network performance (e.g., bandwidth, congestion, andlatency, management), and the required short time-to-build.

The impacts to network communication are several:

-   -   (a) The network industry may not have enough economy of scale to        justify CMOS technology of a smaller footprint;    -   (b) Simpler solutions should be sought to advance network        technology, rather than to create more complex ones and packing        more transistors;    -   (c) Scale-out solutions (i.e., in complement to scale-up        solution) should be sought to solve application problems (e.g.,        big-data, AI, HPC, and data center);    -   (d) The chip port density (i.e., the number of ports in the        traditional sense) can become flat¹; and ¹ Integration of        optical technology to the CMOS device may provide new        opportunity. However, do not expect a very high-radix chip,        which would allow network scalability, to emerge any time soon.    -   (e) Implementation of interfaces with signal rates in excess of        100 G will become increasingly difficult². ² One must think        beyond the aggregation model (e.g., the disaggregation model) to        meet new network challenges.

Historically, high-speed networks have two classes of design space. Inthe first class of design space, HPC and supercomputing networkstypically adopt direct network topologies. In a direct network topology,every switch is connected to servers, as well as other switches in thetopology. Popular topologies include mesh, torus, and hypercube. Thistype of network is highly resource efficient and offers high capacitythrough numerous paths of various lengths between a source anddestination. However, the choice of which path to forward traffic overis ultimately controlled by proprietary protocols (i.e., non-minimumrouting) in switches, NICs, and by the end-host application logic. Thatis, an algorithm or manual configuration is required to achieve routing.Such routing protocols increase the burden on the developer and create atight coupling between applications and the network.

In the second class of design space, data centers scaling-out haveresulted in the development of indirect network topologies, such asfolded-Clos and multi-rooted trees (“fat trees”), in which servers arerestricted to the edges of the network fabric. The interior of thenetwork fabric consists of dedicated switches that are not connected toany servers, but simply route traffic within the network fabric. Datacenter networks of this type thus have a much looser coupling betweenapplications and network topology, placing the burden of path selectionon the network switches themselves. That is to say, based on Internetrouting technology such as BGP (Border Gateway Protocol) routingprotocol. The BGP routing protocol has a complete set of loopprevention, shortest path and optimization mechanisms. However, thereare strict requirements and restrictions on the network topology. Datacenter technology based purely on Internet BGP routing cannoteffectively support multipath with non-shortest path topologies. As aresult data center networks have traditionally relied on fat treetopologies, simple routing and equal cost multipath selection mechanisms(e.g., ECMP). It is precisely because data center routing technology hasrestrictions on the network topology. The benefits to datacenter fromnon-shortest multipath path network topology other than the equal costmultipath topology have not been explored in the past years ofdevelopments of the datacenter technologies.

The BGP and ECMP protocols are not without flaws. ECMP relies on statichashing of flows across a fixed set of shortest equal cost paths to adestination. For hierarchical topologies (e.g., fat tree), ECMP routinghas been largely sufficient when there are no failures. However, evennow direct network topologies (e.g., Dragonfly, HyperX, Slim Fly, BCube,and Flattened Butterfly), which employ paths of different lengths, havenot seen adoption in data centers because of the limitations imposed byboth commodity data center switches and the widespread adoption of ECMProuting in data center networks. ECMP is wasteful of network capacitywhen there is localized congestion or hot-spots, as it ignoresuncongested longer paths. Further, even in hierarchical networks, ECMPmakes it hard to route efficiently in the presence of failures, and whenthe network is no longer completely symmetric, and non-shortest pathsare available for improving network utilization.

FIG. 2 a shows an architecture of a typical state-of-the-art data centernetwork, organized by three layers of switching devices—i.e.,“top-of-rack” (TOR) switches and fabric switches implemented in 96server pods 21-0 to 21-95 and spine switches implemented in 4 spineplanes 20-0 to 20-3—interconnected by interlinks in a fat tree topology.Details of a spine plane is shown in FIG. 2 b where a spine planeconsists of 48 spine switches 22-0 to 22-47 each connecting to 96 serverpods. The connections from all 48 spine switches are grouped into 96interlinks each including a connection from one of spine switches 22-0to 22-47, respectively, for a total of 48 connections per interlink.Details of a server pod is shown in FIG. 2 c, in which a server pod isshown to consist of 48 TOR switches 24-0 to 24-47 and 4 fabric switches23-0 to 23-3, with each TOR switch connected to all 4 fabric switches.Combining the connection information from FIG. 2 b , server pod of FIG.2 c may comprise 4 fabric switches 23-0 to 23-3 each connects one of 4spine planes by interlinks, respectively; each interlink comprising 48connections each connects one of 48 spine switches in a spine plane,respectively. Each TOR switch provides 48×10 G connections in 12×QSFPinterfaces as downlink to connect to servers. An edge pod is shown inFIG. 2 d, details will be given in below.

As shown in FIGS. 2 b and 2 c, and in conjunction with FIG. 2 a, theTOR, fabric and spine layers of switches include: (a) a TOR switch layerconsisting of 96×48 TOR switches which connect the servers in the datacenter and which are equally distributed over 96 “server pods”; (b) aspine switch layer consisting of 4×48 spine switches equally distributedover the 4 “spine planes”; and (c) a fabric layer consisting of 96×4fabric switches, also equally distributed over the 96 server pods. Inaddition, two of the server pods can be converted to two edge pods. FIG.2 d shows an edge pod. As shown in FIG. 2 d, edge pod 250 may comprise 4edge switches 25-0 to 25-3, each connects one of 4 spine planes byinterlinks, respectively; each interlink comprising 48 connections eachconnects one of 48 spine switches in a spine plane, respectively. Eachedge switch may include one or more uplinks that interconnect anexternal network.

Details of an implementation of the server and spine pods are furtherdescribed below in FIG. 2 c, 2 b in relation to FIG. 2 a . Thisconfiguration facilitates modularity by assembling each fabric switchand spine switch in an 8U multi-unit chassis with 96 QSFP ports. Asshown in FIG. 2 c , each TOR switch is implemented by a switch with 16QSFP ports, which allocates 12 QSFP ports to connect to the servers in10G interfaces (i.e., downlinks) and 4 QSFP ports to connect to the fourfabric switches in four 40G interfaces in the same server pod. (In thisdetailed description, a QSFP represents a 40 Gbits/second bandwidth,which can be provided in a single 40G interface or four 10G interfaces,each 40G interface including four receive-transmit pairs of opticalfibers and each 10G interface including a receive-transmit pair ofoptical fibers). The 40G interface between a TOR switch and a fabricswitch is used for both intra-pod and inter-pod data traffic.

Each fabric switch in a server pod is implemented by a 96 QSFP portsswitch, which allocates (i) 48 QSFP ports in 48 40G interfaces with the48 TOR switches in the server pod in a fat tree topology, and (ii) 48QSFP ports in 48 40G interfaces to the 48 spine switches in the singlespine plane the fabric switch is connected.

Each spine switch in a spine plane is also implemented by a 96 QSFPports switch, which provides all 96 QSFP ports in 96 40G interfaces withthe 96 fabric switches connected to the spine plane, one from each ofthe 96 server pods. The data traffic through the spine plane representsinter-pod communications mostly for the server pods.

In the configuration of FIG. 2 a, each server pod includes (i) 384 QSFPtransceivers, half of which are provided to the spine planes and half ofwhich are provided to the network side of the fabric switches, (ii) 192QSFP transceivers provided to the network side of the TOR switches,(iii) 576 transceivers provided to the servers; (iv) 192 optical QSFPcables, (v) 36 application-specific integrated circuits (ASICs), whichimplements the fabric switches and (vi) 48 ASICs, which implements theTOR switches. The ASIC suitable for this application may be, forexample, the Trident-II Ethernet Switch (“Trident II ASIC”). Each spineplane includes 4608 QSFP transceivers, 4608 optical QSFP cables and 432Trident II ASICs.

The implementation of FIG. 2 a provides in practice improved congestionperformance but does not eliminate congestion. The network organizationis based on an aggregation model, intended to improve cost andutilization of communication ports and transmission media under theaggregation model. While this aggregation model may still be valuablefor wide-area networks (e.g., the Internet), recent advances ofsemiconductor technology and economic of scale have called thisaggregation model into question, when applied to local area networks.

SUMMARY

According to one embodiment of the present invention, an interconnect asa switch module (“ICAS” module) comprises n port groups, each portgroups comprising n-1 interfaces, and an interconnecting networkimplementing a full mesh topology where each port group comprising aplurality of interfaces each connects an interface of one of the otherport groups, respectively.

According to one embodiment of the present invention, a stackableswitching device is provided, which includes one or more ICAS modules asdepicted above, a plurality of switching devices, and a stackablerackmount chassis, each ICAS module being connected to the plurality ofswitching devices, such that the ICAS module interconnects at least someinterfaces of at least some port groups of different switching devicesto form a full mesh non-blocking interconnection, while the restinterfaces of the at least some port groups for interconnectingdifferent switching devices are configured as interfaces for uplink. TheICAS module and the switching devices are housed in the stackablerackmount chassis.

One embodiment of the present invention provides a multi-unit switchingdevice, which includes: one or more ICAS modules implemented on a PCB asa circuit, a plurality of switching devices, and a multi-unit rackmountchassis, each ICAS module being connected to the plurality of switchingdevices, such that the ICAS module interconnects at least someinterfaces of at least some port groups of different switching devicesto form a full mesh non-blocking interconnection, while the restinterfaces of the at least some port groups for interconnectingdifferent switching devices are configured as interfaces for uplink. TheICAS module and the switching devices are packaged in the multi-unitrackmount chassis.

According to one embodiment of the present invention, a network pod isdisclosed, which includes: a plurality of first layer switching devices,each having a plurality of interfaces for downlink interfaces configuredto receive and transmit data signals from and to a plurality of servers,and each having a plurality of network side interfaces divided into aplurality of interlinks and a plurality of intralinks, and the interlinkinterfaces being configured to connect to higher layer switchingdevices, and the intralink interfaces of the first layer switchingdevices each being configured and grouped into one or more port groups;and one or more second layer devices of ICAS modules whose interfacesare divided into intralink interfaces and uplink interfaces, and theintralink interfaces of an ICAS module being grouped into port groups toconnect to the corresponding port groups of the first layer switchingdevices, and each port groups of additional ICAS module being connectedto the additional port group of each of the first layer switches, andthe uplink interfaces are configured to connect to the external network.The first layer switching devices and the second layer devices areinterconnected to implement a full mesh network of a predeterminednumber of nodes.

K spine planes each having p interlinks are used to connect p networkpods each having k TOR switches. In a spine plane, k spine switchesinterconnect to a fanout cable transpose rack.

According to one embodiment of the present invention, a fanout cabletranspose rack may include: k first port groups connecting tocorresponding port groups of k spine switches through first plurality offiber optic cables; p second port groups through connecting secondplurality of fiber optic cables to form p interlinks. A plurality offanout cables are used to cross-connect the k first port groups and thep second port groups so that connections from all k spine switches aregrouped into p interlinks, each interlink including one connection fromeach spine switch, and each interlink having a total of k connections.

According to one embodiment of the present invention, a data centernetwork may have a plurality of interfaces for downlink configured toreceive and transmit data signals from and to a plurality of servers,and a plurality of interfaces for uplink configured to connect theInternet or connect another data center network with a similarconfiguration. The data center network may include: a group of networkpods (server pods/ICAS pods), each network pod in the group including:(a) a group of first layer switching devices, providing some interfacesas interfaces for downlink, and having the rest interfaces grouped intoone or more network side port groups; and (b) one or more second layerdevices, configured to interconnect at least some interfaces betweensome port groups of the first layer switching devices, wherein the restinterfaces of the some port groups for interconnecting the first layerswitching devices are configured as interfaces for uplink. The firstlayer switching devices and the second layer devices are interconnectedto implement a full mesh network of a predetermined number of nodes. Thenetwork pod further comprises a group of switch clusters, each includinga group of third layer switching devices, each of which routes aplurality of data signals received from or transmitted to acorresponding first layer switching device in each group of networkpods.

By simplifying the data center network infrastructure and reducinghardware requirement, the present invention addresses the problemsrelating to the power needs for the equipment and their cooling,facility space, the cost of hardware, network performance (e.g.,bandwidth, congestion, and latency, management), and the required shortbuilt time.

The present invention is better understood upon consideration of thedetailed description below in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a illustrates congestion due to hash collision in a fat treenetwork under ECMP.

FIG. 1 b illustrates aggregation congestion in a fat tree networktopology.

FIG. 1 c illustrates congestion due to blocking condition in a fat treenetwork.

FIG. 2 a shows the architecture of a state-of-the-art data centernetwork.

FIG. 2 b shows in detail an implementation of a spine plane of the datacenter network of FIG. 2 a.

FIG. 2 c shows in detail an implementation of a server pod of FIG. 2 ausing four fabric switches to distribute machine-to-machine trafficacross 48 top-of-rack switches.

FIG. 2 d shows in detail an implementation of an edge pod of FIG. 2 ausing four edge switches to provide interfaces for uplink to connect toexternal network.

FIG. 3 illustrates a “full mesh” topology in a network of 9 nodes.

FIG. 4 a shows ICAS module 400, which interconnects 9 nodes, accordingto the full mesh topology of FIG. 3 .

FIG. 4 b illustrates the connectivity between the internal interfacesand the external interfaces of port group7 of the 9-node ICAS module400, in accordance with one embodiment of the present invention.

FIG. 5 a shows ICAS module 500 connecting port group 2 of each of TORswitches 51-0 to 51-8 in a full mesh topology, in accordance with oneembodiment of the present invention.

FIG. 5 b illustrates, in the full mesh topology network 500 of FIG. 5 a,port group 2 of TOR switch 51-1 routing a data packet to port group 2 ofTOR switch 51-7 through internal interface 52-1-7 of port group 50-1 andinternal interface 52-7-1 of port group 50-7 of ICAS2 module 500, inaccordance with one embodiment of the present invention.

FIG. 6 a shows network 600, which is a more compact representation ofthe network of FIG. 5 a.

FIG. 6 b shows network 620, after additional ICAS modules are added tonetwork 600 of FIG. 6 a, so as to provide greater bandwidth and pathdiversity.

FIG. 7 a shows that, in the architecture of the data center of FIG. 2 a,the topology of a server pod may be reduced to a (4, 48) bipartitegraph.

FIG. 7 b shows, as an example, network 720 represented as a (5, 6)bipartite graph.

FIG. 7 c shows the 6-node full mesh graph embedded in the (5, 6)bipartite graph of FIG. 7 b.

FIG. 8 a shows an improved data center network 800, in accordance withone embodiment of the present invention; data center networks 800includes 20 spine planes, providing optional uplinks 801, and 188 serverpods, providing optional uplinks 802, uplinks 801 and 802 connecting toone or more external networks.

FIG. 8 b shows in detail an implementation of modified spine plane 820,having 20 spine switches, providing optional uplink 821 for connectingto an external network.

FIG. 8 c shows in detail an implementation of modified server pod 830 ina (20, 21) fabric/TOR topology, having 20 fabric switches fordistributing machine-to-machine traffic across 20 top-of-rack switches,in accordance with one embodiment of the present invention; the 21^(st)TOR switch is removed from the modified server pod 830 so that theconnections are provided as optional uplink 831 for connecting thefabric switches to an external network.

FIG. 9 a shows ICAS-based data center network 900, achieved by replacingthe server pods of network 800 of FIG. 8 a (e.g., server pod 830 of FIG.8 c ) with ICAS pods 91-0 to 91-197, each ICAS pod being shown ingreater detail in FIG. 9 c , according to one embodiment of the presentinvention; in FIG. 9 a, optional uplinks 901, shared by 20 spine planes,and optional uplinks 902, shared by 188 ICAS pods are provided forconnecting to an external network.

FIG. 9 b shows in detail spine plane 920, which implements one of thespine planes in data center network 900 and which is achieved byintegrating a fanout cable transpose rack into spine plane 820 of FIG. 8b , according to one embodiment of the present invention; the spineswitches in spine plane 920 provide optional uplink 921 for connectingto an external network.

FIG. 9 c shows in detail an implementation of ICAS pod 930, which isachieved by replacing fabric switches 83-0 to 83-19 in server pod 830 ofFIG. 8 c , according to one embodiment of the present invention; eachICAS pod provides 20×10G uplinks 932 for connecting to an externalnetwork.

FIG. 9 d illustrates a spine switch implemented with a single chiphigh-radix (i.e., a high port count) switching integrated circuit; sucha spine switch makes use of the highest port count switching integratedcircuit available at present time.

FIG. 9 e shows a spine switch formed by stacking together 4 switch boxesimplemented with a Trident-II ASICs (96×10G configuration each) and 1ICAS box 953. ICAS box 953 combines 4 ICAS modules 95-0 to 95-3 in one1U chassis. Each ICAS module contains 3 copies of ICAS1X5 configuration.Together the ICAS box 953 provides non-blocking 1:1 subscription ratioto each of the 4 switches 96-0 to 96-3.

FIG. 9 f shows a spine switch of an ICAS-based multi-unit switchingdevice where 4 ICAS-based fabric cards 97-0 to 97-3 get connected in afull mesh topology to switching ASIC's 98-0 to 98-3. Switching ASIC 98-0and 98-1 are housed in line card 973, and switching ASIC's 98-2 and 98-3are housed in line card 974.

To facilitate cross-referencing among the figures and to simplify thedetailed description, like elements are assigned like referencenumerals.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention simplifies the network architecture by eliminatingthe switches in the fabric layer based on a new fabric topology,referred herein as the “interconnect-as-a-switch” (ICAS) topology. TheICAS topology of the present invention is based on the “full mesh”topology. In a full mesh topology, each node is connected to all othernodes. The example of a 9-node full mesh topology is illustrated in FIG.3 . The inherent connectivity of a full mesh network can be exploited toprovide fabric layer switching.

As discussed in further detail below, the ICAS topology enables a datacenter network that is far superior to a network of the fat treetopology used in prior art data center networks. Unlike other networktopologies, the ICAS topology imposes a structure on the network whichreduces congestion in a large extent. According to one embodiment, thepresent invention provides an ICAS module as a component forinterconnecting communicating devices. FIG. 4 a shows ICAS module 400,which interconnects 9 nodes according to the full mesh topology of FIG.3 .

FIG. 4 a shows ICAS module 400 having port groups 40-0 to 40-8 and eachport group providing 8 external interfaces and 8 internal interfaces. InICAS module 400, each of the internal interfaces of a port groupconnects an internal interface of one of the other port groups,respectively. In fact, each port group is connected to every one of theother port groups through exactly one internal interface. In thiscontext, each “interface ” includes a receive-transmit pair of opticalfibers capable of, for example, a 10 Gbits per second data rate. In FIG.4 a , the port groups are indexed as 0-8. Indexes can also be arbitraryunequal values (For example, these 9 port groups can also be indexed as5, 100, 77, 9, 11, 8, 13, 50, and 64). The 8 internal interfaces forthese 9 port groups are indexed according to the indexes of theconnected port groups (For example, the internal interfaces for the 7-thport group are 0, 1, 2, 3, 4, 5, 6 and 8 in the first example; and are5, 100, 77, 9, 11, 8, 13 and 64 in the second example). Furthermore,internal interface j of port group i is connected to internal interfacei of port group j. The external interfaces for each port group of ICASmodule 400 are indexed sequentially as 0-7.

FIG. 4 b illustrates in detail the connectivity between the internalinterfaces and the external interfaces of a port group 7 in ICAS module400, in accordance with the present invention. As shown in FIG. 4 b, inone embodiment, the external interfaces are connected one-to-one to theinternal interfaces sequentially in the index order (For example, forport group 7, external interfaces 42-0 to 42-7 are sequentiallyconnected to internal interfaces 41-0 to 41-6 and 41-8). Therefore, forport group i, external interfaces 0-7 are connected to internalinterfaces 0, . . . , i-1, i+1, . . . , and 8 respectively. Therefore,it can be easily seen that any pair of port groups x and y are connectedthrough internal interface x of port group y and internal interface y ofport group x. This indexing scheme allows an external switching deviceto assign routes for data packets using the internal interface indicesof the source port group and destination port group. No congestioncondition (e.g., due to hash collision, aggregation model, orstrict-sense blocking) can occur between any pair of port groups.

The internal interconnection between the port groups of the ICAS modulecan be realized via an optical media to achieve a full mesh structure.The optical media may be an optical fiber and/or 3D MEMS. The 3D MEMSuses a controllable micro-mirror to create an optical path to achieve afull mesh structure. In both of these implementations MPO connectors areused. Alternatively, the ICAS module may also be electricallyimplemented using circuits. In this manner, the port groups of the ICASmodule are soldered or crimped onto a PCB using connectors that supporthigh-speed differential signals and impedance matching. Theinterconnection between the port groups is implemented using a copperdifferential pair on the PCB. Since signal losses significantly varybetween different grades of high-speed differential connectors andbetween copper differential pairs on different grades of PCBs, an activechip is usually added at the back end of the connector to restore andenhance the signal to increase the signal transmission distance on thePCB. Housing the ICAS module in a 1U to multi-U rackmount chassis willform a 1U to multi-U interconnection device. The ICAS-basedinterconnection devices are then interconnected with switching devicesto form a full mesh non-blocking network. This novel network will beexplained in detail hereunder in a plurality of embodiments. When theICAS module of the 1U to multi-U interconnection device is opticallyimplemented (based on optical fiber and 3D MEMS), MPO-MPO cables areused to connect the ICAS-based interconnection devices and the switchingdevices. When the ICAS module of the 1U to multi-U interconnectiondevice is electrically implemented as circuits (based on PCB+chip), DACdirect connection cables or AOC active optical cables are used toconnect the ICAS-based interconnection devices and the switchingdevices.

As switching in ICAS module 400 is achieved passively by itsconnectivity, no power is dissipated in performing the switchingfunction. Typical port group-to-port group delay through an ICAS passiveswitch is around 10 ns (e.g., 5 ns/meter, for an optical fiber), makingit very desirable for a data center application, or for big data, AI andHPC environments.

The indexing scheme of external-to-internal connectivity in ICAS module400 of FIG. 4 a is summarized in Table 2 below:

TABLE 2 Index of External Interface ICAS Port Group 0 1 2 3 4 5 6 7 0 12 3 4 5 6 7 8 1 0 2 3 4 5 6 7 8 2 0 1 3 4 5 6 7 8 3 0 1 2 4 5 6 7 8 4 01 2 3 5 6 7 8 5 0 1 2 3 4 6 7 8 6 0 1 2 3 4 5 7 8 7 0 1 2 3 4 5 6 8 8 01 2 3 4 5 6 7

FIG. 5 a shows network 500, in which ICAS module 510 and port group 2 ofeach of TOR switches 51-0 to 51-8 interconnects in a full mesh topology,in accordance with one embodiment of the present invention.

As illustrated in FIG. 5 b, in an ICAS module 510 in the full meshtopology network 500 of FIG. 5 a, port group 51-1 of TOR switch 1 routesa data packet to port group 51-7 of TOR switch 7 through externalinterface 53-1-6 and internal interface 52-1-7 of port group 50-1 ofICAS module 510, and internal interface 52-7-1 and external interface53-7-1 of port group 50-7 of ICAS module 510, in accordance with oneembodiment of the present invention. As shown in FIG. 5 b , TOR switch51-1, which is connected to port group 50-1 of ICAS module 510, receivesa data packet with a destination reachable through internal port group52-1-7 of ICAS module 510. TOR switch 51-1 has a port group thatincludes 8 interfaces 54-1-0 to 54-1-7 (provided as two QSFP ports)mapping one-to-one to external interfaces 53-1-0 to 53-1-7 of port group50-1 of ICAS module 510, which in turn maps one-to-one to internalinterfaces 52-1-0, 52-1-2 to 52-1-8 in sequential order of port group50-1 of ICAS module 510. TOR switch 51-7 has a port group that includes8 interfaces 54-7-0 to 54-7-7 (provided as two QSFP ports) mappingone-to-one to external interfaces 53-7-0 to 53-7-7 of port group 50-7 ofICAS module 510, which in turn maps one-to-one to internal interfaces52-7-0 to 52-7-6 and 52-7-8 in sequential order of port group 50-7 ofICAS module 510. Each interface in a TOR switch port may be a 10Ginterface, for example. As port groups 50-1 and 50-7 of ICAS module 510are connected through the port groups' corresponding internal interfaces52-1-7 and 52-7-1, TOR switch 51-1 sends the data packet through itsinterface 54-1-6 to external interface 53-1-6 of ICAS module 510. Sincethe connectivity in the ICAS module 510 adopts a full mesh topology, thedata packet is routed to external interface 53-7-1 of ICAS module 510.

In full mesh topology network 500, the interfaces of each TOR switch isregrouped into port groups, such that each port group contains 8interfaces. To illustrate this arrangement, port group 2 from each TORswitch connects to ICAS module 510. As each TOR switch has a dedicatedpath through ICAS module 510 to each of the other TOR switches, nocongestion can result from two or more flows from different sourceswitches being routed to the same port of destination switch (the“Single-Destination-Multiple-Source Traffic Aggregation” case). In thatcase, for example, when TOR switches 51-0 to 51-8 each have a 10-G dataflow that has TOR switch 51-0 as destination, all the flows would berouted on paths through respective interfaces. Table 3 summarizes theseparate designated paths:

TABLE 3 ICAS ICAS Source Destination Source Internal InternalDestination T1.p2.c0 ↔ ICAS2.pl.c0 ↔ ICAS2.p0.c1 ↔ T0.p2.c0 T2.p2.c0 ↔ICAS2.p2.c0 ↔ ICAS2.p0.c2 ↔ T0.p2.c1 T3.p2.c0 ↔ ICAS2.p3.c0 ↔ICAS2.p0.c3 ↔ T0.p2.c2 T4.p2.c0 ↔ ICAS2.p4.c0 ↔ ICAS2.p0.c4 ↔ T0.p2.c3T5.p2.c0 ↔ ICAS2.p5.c0 ↔ ICAS2.p0.c5 ↔ T0.p2.c4 T6.p2.c0 ↔ ICAS2.p6.c0 ↔ICAS2.p0.c6 ↔ T0.p2.c5 T7.p2.c0 ↔ ICAS2.p7.c0 ↔ ICAS2.p0.c7 ↔ T0.p2.c6T8.p2.c0 ↔ ICAS2.p8.c0 ↔ ICAS2.p0.c8 ↔ T0.p2.c7

In other words, in Table 3, the single-connection data between firstlayer switch i connected to the port group with index i and first layerswitch j connected to the port group with index j is directlytransmitted through the interface with index j of the port group withindex i and the interface with index i of the port group with index j.

In Table 3 (as well as in all Tables herein), the switch source and theswitch destination are each specified by 3 values: Ti.p_(j).c_(k), whereT_(i) is the TOR switch with index i, p_(j) is the port group with indexj and c_(k) is the interface with index k. Likewise, the sourceinterface and destination interface in ICAS module 500 are also eachspecified by 3 values: ICASj.p_(i).c_(k), where ICASj is the ICAS modulewith index j, p_(i) is the port group with index i and c_(k) is theinternal or external interface with index k.

An ICAS-based network is customarily allocated so that when its portgroups are connected to port group i from all TOR switches the ICAS willbe labeled as ICASi with index i.

Congestion can also be avoided in full mesh topology network 500 with asuitable routing method, even when a source switch receives a largeburst of aggregated data (e.g., 80 Gbits per second) from all itsconnected servers to be routed to the same destination switch (the“Port-to-Port Traffic Aggregation” case). In this case, it is helpful toimagine the TOR switches as consisting of two groups: the source switchi and the rest of the switches 0 to i−1, i+1 to 8. The rest of theswitches are herein collectively referred to as the “fabric group”.Suppose TOR switch 51-1 receives 80 Gbits per second (e.g., 8 10G flows)from all its connected servers all designating to destination TOR switch51-0. The routing method for the Port-to-Port Traffic Aggregation caseallocates the aggregated traffic to its 8 10G interfaces with port group51-1 as in FIG. 5 a , such that the data packets in each 10G interfaceis routed to a separate TOR switch in the fabric group (Table 4A):

TABLE 4A ICAS ICAS Source Destination Source Internal InternalDestination T1.p2.c0 ↔ ICAS2.p1.c0 ↔ ICAS2.p0.c1 ↔ T0.p2.c0 T1.p2.c1 ↔ICAS2.p1.c2 ↔ ICAS2.p2.c1 ↔ T2.p2.c1 T1.p2.c2 ↔ ICAS2.p1.c3 ↔ICAS2.p3.c1 ↔ T3.p2.c1 T1.p2.c3 ↔ ICAS2.p1.c4 ↔ ICAS2.p4.c1 ↔ T4.p2.c1T1.p2.c4 ↔ ICAS2.p1.c5 ↔ ICAS2.p5.c1 ↔ T5.p2.c1 T1.p2.c5 ↔ ICAS2.p1.c6 ↔ICAS2.p6.c1 ↔ T6.p2.c1 T1.p2.c6 ↔ ICAS2.p1.c7 ↔ ICAS2.p7.c1 ↔ T7.p2.c1T1.p2.c7 ↔ ICAS2.p1.c8 ↔ ICAS2.p8.c1 ↔ T8.p2.c1

Note that the data routed to TOR switch 51-0 has arrived at itsdesignation and therefore would not be routed further. Each TOR switchin the fabric group, other than TOR switch 51-0, then allocates itsinterface 0 for forwarding its received data to TOR switch 51-0 (Table4B):

TABLE 4B ICAS ICAS Source Destination Source Internal InternalDestination — ↔ — ↔ — ↔ — T2.p2.c0 ↔ ICAS2.p2.c0 ↔ ICAS2.p0.c2 ↔T0.p2.c1 T3.p2.c0 ↔ ICAS2.p3.c0 ↔ ICAS2.p0.c3 ↔ T0.p2.c2 T4.p2.c0 ↔ICAS2.p4.c0 ↔ ICAS2.p0.c4 ↔ T0.p2.c3 T5.p2.c0 ↔ ICAS2.p5.c0 ↔ICAS2.p0.c5 ↔ T0.p2.c4 T6.p2.c0 ↔ ICAS2.p6.c0 ↔ ICAS2.p0.c6 ↔ T0.p2.c5T7.p2.c0 ↔ ICAS2.p7.c0 ↔ ICAS2.p0.c7 ↔ T0.p2.c6 T8.p2.c0 ↔ ICAS2.p8.c0 ↔ICAS2.p0.c8 ↔ T0.p2.c7

In other words, at least one multi-connection data between the firstlayer switch i connected to the port group indexed i and the first layerswitch j connected to the port group indexed j is transmitted throughthe first layer switches connected to at least one of the port groupsother than the port group with source index. The multi-connection dataarriving at the destination switch will cease to be further routed andtransmitted.

To put it more precisely, the multi-connection data transmissionoccurring between first layer switch i connected to the port group withindex i and first layer switch j connected to the port group with indexj includes the transmissions includes: as in Table 4A, the first layerswitch i is connected, via a plurality of interfaces of the port groupwith a plurality of index i, to a plurality of first layer switches witha plurality of corresponding indexes for transmission; as in Table 4B, aplurality of the first layer switches with the indexes as shown areconnected, via interfaces with index j of the port groups, to theinterfaces with the indexes as shown of the port groups with index j ofthe first layer switches for transmission; those transmissions thatarrive at a destination switch will stop routing.

Thus, the full mesh topology network of the present invention providesperformance that is in stark contrast to prior art network topologies(e.g., fat tree), in which congestions in the fabric switch cannot beavoided under Single-Destination-Multiple-Source Traffic Aggregation andPort-to-Port Traffic Aggregation cases.

Also, as discussed above, when TOR switches 51-0 to 51-8 abide by therule m≥2n-2, where m is the number of network-side interfaces (e.g., theinterfaces with a port group in ICAS module 500) and n is the number ofthe TOR switch's input interfaces (e.g., interfaces to the serverswithin the data center), a strict blocking condition is avoided. Inother words, a static path is available between any pair of inputinterfaces under any traffic condition. Avoiding such a blockingcondition is essential in a circuit-switched network, but is notnecessarily significant in a flow-based switched network.

In the full mesh topology network 500 of FIG. 5 a , each port group with8 interfaces of ICAS module 500 connects to a port group with 8interfaces (e.g., 8 10-G interfaces) of a corresponding TOR switch. Fullmesh topology network 500 of FIG. 5 a may be redrawn in a more compactform in FIG. 6 a , with a slight modification. FIG. 6 a illustratesICAS2 module 60-2 interconnecting to port group 2 of each of TORswitches 61-0 to 61-8. In FIG. 6 a , the interfaces between port group 2of TOR switch 61-0 and port group 0 of ICAS module 60-2 (now labeled‘ICAS2’) are represented as a single line (e.g., the single line betweenport group 2 of TOR switch 61-0 and port group 0 of ICAS module 60-2).Such a line, of course, represents all 8 eight interfaces between theTOR switch and a corresponding port group in ICAS module 60-2. This isexactly the case in FIG. 6 b where each TOR switch 63-0 to 63-8 is shownalso to have 4 port groups, to allow configuring network 620 of FIG. 6b, where three additional ICAS modules 62-0, 62-1 and 62-3 in additionto 62-2 and corresponding interfaces are added to network 600 of FIG. 6a.

In full mesh topology network 500, uniform traffic may be spread out tothe fabric group and then forwarded to its destination. In network 620of FIG. 6 b, the additional ICAS modules may be used to provide greaterbandwidth. So long as the additional port groups are available in theTOR switches, additional ICAS modules may be added to the network toincrease path diversity and bandwidth.

The inventor of the present invention investigated in detail thesimilarities and the differences between the full mesh topology of thepresent invention and other network topologies, such as the fat treetopology in the data center network of FIGS. 2 a. The inventor firstobserves that, in the architecture of the data center network of FIG. 2a , the fat tree network represented in a server pod (the “fabric/TORtopology”) can be reduced to a (4, 48) bipartite graph, so long as thefabric switches merely perform an interconnect function for trafficoriginated among the TOR switches. This (4, 48) bipartite graph is shownin FIG. 7 a . In FIG. 7 a , the upper set of nodes, nodes 0-3 (“fabricnodes”) 70-0 to 70-3, represent the four fabric switches in the serverpod of FIG. 2 a and the lower set of 48 nodes (i.e., leaf 0-47), labeled71-0 to 71-47, represent the 48 TOR switches in a server pod of FIG. 2a.

The inventor discovered that an n-node full mesh graph is embedded in afabric-leaf network represented by a bipartite graph with (n-1, n) nodes(i.e., a network with n-1 fabric nodes and n TOR switch leaves). FIG. 7b shows, as an example, a (5, 6) bipartite graph with 5 nodes 72-0 to72-4 and 6 leaves 73-0 to 73-5. FIG. 7 c shows the 6-node full meshgraph 740 with 6 nodes 74-0 to 74-5 embedded in the (5, 6) bipartitegraph of FIG. 7 b.

This discovery leads to the following rather profound results:

-   -   (a) An n-node full mesh graph is embedded in an (n-1,        n)-bipartite graph; and the (n-1, n) bipartite graph and the        data center Fabric/TOR topology have similar connectivity        characteristics;    -   (b) A network in the (n-1, n) Fabric/TOR topology (i.e., with        n-1 fabric switches and n TOR switches) can operate in same        connectivity characteristics as a network with full mesh        topology (e.g., network 500 of FIG. 5 a );    -   (c) Fabric switches are unnecessary in an (n-1, n) Fabric/TOR        topology network, as the fabric switches merely performs        interconnecting function among the TOR switches (i.e., these        fabric switches can be replaced by direct connectivity among TOR        switches);    -   (d) A data center network based on a fat tree topology (e.g.,        the Fabric/TOR topology) can be improved significantly using        ICAS modules.

In the following, a data center network that incorporates ICAS modulesin place of fabric switches may be referred to as an “ICAS-based” datacenter network. An ICAS-based data center network has the followingadvantages:

-   -   (a) less costly, as fabric switches are not used;    -   (b) lower power consumption, as ICAS modules are passive;    -   (c) less congestion;    -   (d) lower latency;    -   (e) effectively less network layers (2 hops less for inter-pod        traffic; 1 hop less for intra-pod traffic);    -   (f) greater scalability as a data center network.

These results may be advantageously used to improve typicalstate-of-the-art data center networks. FIG. 8 a shows an improved datacenter network 800, in accordance with one embodiment of the presentinvention. Data center network 800 uses the same types of components asthe data center network of FIG. 2 a (i.e., spine switches, fabricswitches and TOR switches), except that the number of fabric switchesare increased to one less than the number of TOR switches (FIG. 8 cshows equal number of fabric switches and TOR switches because one ofthe TOR switch, the 21^(st) TOR switch, is removed so that the 20interfaces connected to it from the 20 fabric switches are provided asuplink to connect to external network).

FIG. 8 a shows the architecture of an improved data center network,organized by three layers of switching devices—i.e., “top-of-rack” (TOR)switches and fabric switches implemented in 188 server pods 81-0 to81-187 and spine switches implemented in 20 spine planes 80-0 to80-19—interconnected by interlinks in a fat tree topology. An interlinkrefers to the network connections between a server pod and a spineplane. For example, interlink k of each of the 188 server pods isconnected to spine plane k; interlink p of each of the 20 spine planesis connected to server pod p. The 20 spine planes each provide anoptional uplink (e.g. uplink 801) and the 188 server pods each providean optional uplink (e.g., uplink 802) for connection to one or moreexternal networks. In this example, to allow comparison, the numbers ofserver pods and spine plane are chosen so that the improved data centernetwork 800 and the state-of-the-art data center network 200 have thesame network characteristics (2.2 Pbps total server-side bandwidth; 3:1oversubscription ratio—server-side to network-side bandwidth ratio;Trident-II ASIC). Other configurations of the improved data centernetwork are also possible, for instance, 32-TOR server pod or 48-TORserver pod but with higher radix switching silicon than the Trident-IIASIC.

Details of a spine plane of FIG. 8 a are shown in FIG. 8 b . In FIG. 8 b, spine plane 820 consists of 20 spine switches 82-0 to 82-19 eachconnecting to 188 server pods. The connections from all 20 spineswitches are grouped into 188 interlinks, with each interlink includinga connection from each spine switch 82-0 to 82-19, for a total of 20connections per interlink.

Details of a server pod of FIG. 8 a are shown in FIG. 8 c . In FIG. 8 c, the network-side connection (as opposed to the server-side connection)of the server pod is separated into intra-pod links and inter-pod links(i.e., the interlinks). The two types of links are made independent fromeach other. The intra-pod region 832 consists of the intra-pod links,the 20 TOR switches 84-0 to 84-19 and the 20 fabric switches 83-0 to83-19 interconnected by the intra-pod links in a fat tree topology. Forexample, connection k in each of the 20 TOR switches is connected tofabric switch k; connection p of each of the 20 fabric switches isconnected to TOR switch p. 20 fabric switches each provide an optionaluplink (e.g., uplink 831) to connect to an external network. Theinter-pod region consists of the inter-pod links (i.e., the interlinks)and 20 TOR switches 84-0 to 84-19 on the interlink side. Each interlinkprovides 20 10G connections to connect to all 20 spine switches on thesame spine plane. Each server pod includes a total of 20 links. Forexample, interlink k of each of the 188 TOR switches across the 188server pods are connected to spine plane k; interlink p of each of the20 spine planes are connected to server pod p. Each TOR switch provides48×10G connections in 12×QSFP ports as downlink to connect to servers.

The data traffic through the fabric switches is primarily limited tointra-pod. The TOR switches now route both the intra-pod traffic as wellas inter-pod traffic and are more complex. The independent link typesachieve massive scalability in data center network implementations.(Additional independent links provided from higher radix switching ASICmay be created to achieve larger scale of connectivity objectives).Additionally, data center network 800 incorporates the full meshtopology concept (without physically incorporating an ICAS module) toremove redundant network devices and allow the use of innovativeswitching methods, in order to achieve a “lean and mean” data centerfabric with improved data traffic characteristics.

As shown in FIG. 8 c , FIG. 8 b and FIG. 8 a , data center network 800includes 20×188 TOR switches and 20×188 fabric switches equallydistributed over 188 server pods, and 20×20 spine switches equallydistributed over 20 spine planes. In FIG. 8 a , each TOR switch has 10010G-connections (i.e., 25 QSFPs of bandwidth in 10G mode), of which 6010G-connections are provided server-side and 40 10G-connections areprovided network-side. (Among the network-side connections 2010G-connections are used for intra-pod traffic and 20 10G-connectionsare used for inter-pod traffic). In each server pod, fabric switches83-0 to 83-19 each include 21 10G-connections, of which 2010G-connections are allocated to connect with a 10G-connection in eachof TOR switches 84-0 to 84-19, and the rest being converted to provideas uplink to connect to external network. In this manner, fabricswitches 83-0 to 83-19 support the intra-pod region data traffic and theuplinks in the server pod by a 21-node full mesh topology (with theuplinks of fabrics switches 0-19 collectively seen as one node). Using asuitable routing algorithm, such as any of those described above inconjunction with Single-Source-Multiple-Destination Traffic Aggregationand Port-to-Port Traffic Aggregation, network congestion can beeliminated from all fabric switches.

As the network in the intra-pod region of each server pod can operate inthe same connectivity characteristics as a full mesh topology network,all the 20 fabric switches of the server pod may be replaced by an ICASmodule. ICAS-based data center network 900, resulting from substitutingfabric switches 83-0 to 83-19 of data center network 800, is shown inFIG. 9 a . To distinguish from the server pod of data center network800, a server pod with its fabric switches replaced by an ICAS module isreferred to as an “ICAS pod.”

FIG. 9 a shows the architecture of an ICAS-based data center network,organized by three layers of devices—i.e., “top-of-rack” (TOR) switches,ICAS module implemented in 188 server pods 91-0 to 91-187 and spineswitches implemented in 20 spine planes for 90-0 to 90-19—interconnectedby interlinks in a fat tree topology. 20 spine planes provide optionaluplinks 901 and 188 ICAS pods provide optional 188×20×10G uplinks 902for connecting to an external network. The number of network devices inthe data center network should be interpreted as illustrative only.

Details of a spine plane of FIG. 9 a are shown in FIG. 9 b according toone embodiment. In FIG. 9 b , spine plane 920 includes 20 spine switches92-0 to 92-19 and a fanout cable transpose rack 921. The fanout cabletranspose rack contains: k first port groups 923 are connected tocorresponding port groups of k spine switches through a plurality offirst MPO-MPO fiber cables, where each first port group including ┌p/m┐first MPO adapters, and each first MPO adapter including m interfaces(where each interface includes one transmit fiber channel and onereceive fiber channel), and a plurality of first MPO fiber adapters fromthe k port groups 923 are connect to LC optical fiber adapter mountingpanel 922 through a plurality of first MPO-LC fanout fiber cables ,where k=20, p=188, m=4, and ┌┐ is a ceiling function; the fanout cabletranspose rack 921 includes p second port groups 924 that are connectedto a plurality of second MPO-MPO fiber cables to form interlinks 99-0 to99-187, each second port group contains ┌k/m┐ second MPO fiber adapters,each of which includes m interfaces (where each interface includes onetransmit fiber channel and one receive fiber channel), and a pluralityof second MPO fiber adapters from the p port groups 924 are connected toLC optical fiber adapter mounting panel 922 through a plurality of thesecond MPO-LC fanout cables; a plurality of first MPO-LC fanout fibercables cross-connect a plurality of second MPO-LC fanout fiber cables onthe LC fiber adapter mounting panel 922, through cross-connection, allconnections from k spine switches 92-0 to 92-19 are reorganized into pinterlinks 99-0 to 99-187, each interlink includes one connection fromeach of the spine switches 92-0 to 92-19, each interlink contains kconnections in total.

That is, on one side of the fanout cable transpose rack 921 is k firstport groups 923, each first port group has ┌p/m┐ of first MPO adapters,where ┌┐ is a ceiling function, each port groups connects to acorresponding port group of a spine switch through the ┌p/m┐ firstMPO-MPO cables. On the other side of the fanout cable transpose rack 921is p second port groups 924, each second port group has ┌k/m┐ of secondMPO adapters, where ┌┐ is an ceiling function, each port group connectsto 5 second MPO-MPO cables to form an interlink to the ICAS pod.

As pointed out earlier in this detailed description, thestate-of-the-art data centers and switch silicon are designed with 4interfaces (TX, RX) at 10 Gb/s or 25 Gb/s each per port in mind.Switching devices are interconnected at the connection level inICAS-based data center. In such a configuration, a QSFP cable coming outfrom a QSFP transceiver is separated into 4 interfaces, and 4 interfacesfrom different QSFP transceivers are combined in a QSFP cable forconnecting to another QSFP transceiver. Also, a spine plane mayinterconnect a large and varying number of ICAS pods (e.g., in thehundreds) because of the scalability of an ICAS-based data centernetwork. Such a cabling scheme is more suitable to be organized in afanout cable transpose rack (e.g., fanout cable transpose rack 921),which may be one or multiple racks and be integrated into the spineplanes. Specifically, the spine switches and the TOR switches may eachconnect to the fanout cable transpose rack with QSFP straight cables.Such an arrangement simplifies the cabling in a data center. FIG. 9 billustrates such an arrangement for data center network 900 of FIG. 9 a.

In the embodiment shown in FIG. 9 b , the first and the second opticalfiber adapters are MPO adapters, the first and the second cables areMPO-MPO cables, the first and the second fanout cables are MPO-LC fanoutcables, the mounting panel is LC optical fiber adapter mounting panel.One skilled in the art would understand that different types of opticalfiber adapters/cable/optical fiber adapter mounting panel may also beused, such as FC, SC, LC, and MU.

Details of an ICAS pod of FIG. 9 a are shown in FIG. 9 c . In FIG. 9 c ,the network side interface (as opposed to the server-side interface) ofan ICAS pod is divided into intra-pod links (i.e. intralinks) andinter-pod links (i.e., interlinks) and the two types of links are madeindependent from each other. The intra-pod region consists of intralinksbetween the 20 TOR switches 93-0 to 93-10 19 and ICAS module 931,interconnected by 10G connections in a full mesh topology. Each ICASmodule may provide 20 10G uplinks 932 to connect to one or more externalnetworks. The inter-pod region consists of interlinks. ICAS pod maycomprise 20 TOR switches 93-0 to 93-19 each connects one of 20 spineplanes by interlinks, respectively; each interline comprising 20connections each connects one of 20 spine switches in a spine plane,respectively. For example, interlink k of each of 188 TOR switchesacross the 188 ICAS pods is connected to spine plane k; interlink p ofeach of the 20 spine planes is connected to server pod p. Each TORswitch provides 60×10G interfaces in 15×QSFP ports as a downlink forconnecting to servers.

The data traffic through the ICAS module is primarily limited tointra-pod. The TOR switches now perform routing for the intra-podtraffic as well as inter-pod traffic and are more complex. Theindependent link types achieve massive scalability in data centernetwork implementations. (Additional independent link provided fromhigher radix switching ASIC may be created to achieve a larger scale ofconnectivity objectives).

As shown in FIG. 9 c , FIG. 9 b and FIG. 9 a , each TOR switch allocates20×10G-interfaces (5×QSFPs in 10G mode) to connect to its associatedICAS module (e.g., ICAS module 931) to support intra-pod traffic, and 5QSFPs in 10G mode (20 10G-interfaces) to connect to the fiber transposerack to support inter-pod traffic. As shown in FIG. 9 c , each ICAS podincludes 20×5 QSFP transceivers for intra-pod traffic, connected by 100QSFP straight cables, and 20×15 QSFP (10G mode) transceivers for servertraffic, for a total 500 QSFP transceivers. The 20 TOR switches in anICAS pod may be implemented by 20 Trident II ASICs. Although 20 TORswitches are shown in each ICAS pod in FIG. 9 c , the ICAS module isscalable to connect up to 48 TOR switches in an ICAS pod (based on32×QSFP Trident-II+switch ASIC).

Together, the ICAS pods and the spine planes form a modular networktopology capable of accommodating hundreds of thousands of 10G-connectedservers, scaling to multi-petabit bisection bandwidth, and covering adata center with congestion improved and non-oversubscribed rack-to-rackperformance.

According to one embodiment of the present invention, a spine switch canbe implemented using a high-radix (e.g., 240×10G) single chip switchingdevice, as shown in FIG. 9 d . Single-chip implementation saves the costof extra transceivers, cables, rack space, latency and power consumptionthan multi-unit (rack unit) chassis-based switching device and stackableswitching device implementations. The disadvantage of the single-chipspine switch approach is its network scalability, which limits thesystem to 240 ICAS pods at this time. As mentioned above, thesemiconductor implementation limits the scale of a high-radix switchingintegrated circuit.

To overcome the limitation on the port count of the silicon chip, one ormore 1U to multi-U rackmount chassis each packaged with one or more ICASmodules, and a plurality of 1U rackmount chassis each packaged with oneor more switching devices, can be stacked up in one or more racks,interconnected, to form a higher-radix (i.e. high network port count)stackable spine switching device (e.g., ICAS-based stackable switchingdevice). Each ICAS module is connected to the plurality of switchingdevices, such that the ICAS module interconnects at least someinterfaces of at least some port groups of different switching devicesto form a full mesh non-blocking interconnection. The interfaces of therest of the at least some port groups for interconnecting differentswitching devices are configured as an uplink. When theICAS-module-based 1U to multi-U rackmount chassis are opticallyimplemented (based on optical fiber and 3D MEMS), MPO-MPO cables may beused to connect the ICAS-based interconnection devices and the switchingdevices. When the ICAS-module-based 1U to multi-U rackmount chassis areelectrically implemented as circuits (based on PCB+chip), DAC directconnection cables or AOC active optical cables may be used to connectthe ICAS-based interconnection devices and the switching devices.

Details of an ICAS-based stackable switching device 950 are shown inFIG. 9 e . FIG. 9 e shows ICAS modules 95-0 to 95-3 each connected in afull mesh topology to switches 96-0 to 96-3. In one embodiment, 4Trident-II ASIC-based switches 96-0 to 96-3, each having a switchingbandwidth of 24 QSFPs in 10G mode provided in 1:1 subscription ratio,and an ICAS box 953 integrating 4 ICAS modules 95-0 to 95-3 in one 1Uchassis and each ICAS module containing 3 duplicate copies of ICAS1X5sub-modules and each sub-module providing 4×10G of uplink 951 may beused to builds a stackable spine switch, as shown in FIG. 9 e . The 4switches 96-0 to 96-3 provide ports 952 of 1.92 Tbps of bandwidth toconnect to servers. The ICAS-based stackable switching device 950provides total uplink bandwidth of 480 Gb/s (4×3×40 Gb/s) to connect toexternal network, facilitates non-blocking 1:1 subscription ratio andprovides full mesh non-blocking interconnect with a total of 1.92 Tbpsof switching bandwidth.

ICAS-based stackable switching device has the benefits of improvednetwork congestion, saving the costs, power consumption and spacesavings than the switching devices implemented in the state of the artdata center. As shown in the “ ICAS+Stackable Chassis” column of Table5, data center with ICAS and ICAS-based stackable switching deviceperforms remarkably on data center network with total switching ASICsaving by 53.5%, total power consumption saving by 26.0%, total spacesaving by 25.6% and much improved network congestion. However total QSFPtransceiver usage is increased by 2.3%.

The above stackable switching device is for illustrative purpose. Aperson experienced in the art can easily expand the scalability of thestackable switching device and should not be limited as in theillustration.

The stackable switching device addresses the insufficiency in the numberof ports of network switching chip, thus making possible a flexiblenetwork configuration. However, a considerable number of connectingcables and conversion modules have to be used to interconnect theICAS-based interconnection devices and the switching devices. To furtherreduce the use of cables and conversion modules, ICAS modules and switchchips can be electronically interconnected using a PCB and connectors,which is exactly how the multi-unit switching device is structured.Specifically, the ICAS module of the ICAS-based multi-unit switchingdevice is electrically implemented as circuits, and the port groups ofthe ICAS module are soldered or crimped onto a PCB using connectors thatsupport high-speed differential signals and impedance matching. Theinterconnection between the internal port groups is realized using acopper differential pair on the PCB. Since signal losses varysignificantly between different grades of high-speed differentialconnectors and between copper differential pairs on different grades ofPCBs, an active chip can be added at the back end of the connector torestore and enhance the signal to increase the signal transmissiondistance on the PCB. The ICAS module of the ICAS-based multi-unitswitching device may be implemented on a PCB called a fabric card, or ona PCB called a backplane. The copper differential pair on the PCBinterconnects the high-speed differential connector on the PCB to form afull mesh connectivity in the ICAS architecture. The switch chips andrelated circuits are soldered onto a PCB called a line card, which isequipped with a high-speed differential connector docking to the adapteron the fabric card. A multi-U chassis of the ICAS-based multi-unitswitching device includes a plurality of ICAS fabric cards, a pluralityof line cards, and one or two MCU- or CPU-based control cards, one ormore power modules and cooling fan modules. “Rack unit” (“RU” or “U” forshort) measures the height of a data center chassis, equal to 1.75inches. A complete rack is 48U (48 rack units) in height.

One embodiment of the present invention also provides a chassis-basedmulti-unit (rack unit) switching device. A multi-unit chassis switchingdevice groups multiple switch ICs onto multiple line cards.Chassis-based multi-unit switching equipment interconnects with linecards, control cards, and CPU cards via PCB-based network cards orbackplanes, which saves the cost of transceivers, fiber optic cable andrack space required for interconnection.

Details of an ICAS-based multi-unit chassis switching device 970 areshown in FIG. 9 f . FIG. 9 f shows 4 ICAS-based fabric cards 97-0 to97-3 interconnected in a full mesh topology to switching ASIC's 98-0 to98- 3. Switching ASIC 98-0 and 98-1 are housed in line card 973, andswitching ASIC's 98-2 and 98-3 are housed in line card 974. Line cards973 and 974 are connected through high speed PCB (printed circuit board)connectors to fabric cards 97-0 to 97-3. In one embodiment, 4 Trident-IIASIC-based switches 98-0 to 98-3, each having a switching bandwidth of24 QSFPs in 10G mode provided in 1:1 subscription ratio, and 4ICAS-based fabric cards 97-0 to 97-3 containing 3 duplicate copies ofICAS1X5 sub-modules and each sub-module providing 4×10G of uplink 971may be used to builds a multi-unit chassis switch, as shown in FIG. 9 fTwo line cards provide data ports 972 of total 1.92 Tbps of bandwidth toconnect to servers. ICAS-based multi-unit chassis switching device 970provides total uplink bandwidth of 480 Gb/s (4×3×40 Gb/s) to connect toexternal network, facilitates full mesh non-blocking 1:1 subscriptionratio interconnect with a total of 1.92 Tbps of switching bandwidth.

Multi-unit chassis-based switching device with fabric cards that areICAS-based full mesh topology has the benefits of improved networkcongestion, saving the costs and power consumption than that ofASIC-based fabric cards implementation with fat tree topology. As shownin the “ICAS+Multi-unit Chassis” column of Table 5, data center withICAS and ICAS-based multi-unit chassis-based switching device performsremarkably on data center network with total QSFP transceiver saving by12.6%, total switching ASIC saving by 53.5%, total power consumptionsaving by 32.7%, total space saving by 29.95% and much improved networkcongestion.

The above multi-unit chassis switching device is for illustrativepurpose. A person experienced in the art can easily expand thescalability of the multi-unit chassis switching device and should not belimited as in the illustration.

The multi-unit chassis-based switching device has the disadvantage of amuch longer development time and a higher cost to manufacture due to itssystem complexity, and is also limited overall by the form factor of themulti-unit chassis. The multi-unit chassis-based switching device,though provides a much larger port count than the single-chip switchingdevice. Although the stackable switching device requires additionaltransceivers and cables than that of the multi-unit chassis-basedapproach, the stackable switching device approach has the advantage ofgreater manageability in the internal network interconnection, virtuallyunlimited scalability, and requires significantly less time forassembling a much larger switching device.

The material required for (i) the data center networks of FIG. 2 a ,using state of the art multi-unit switching device (“Fat tree+Multi-unitChassis”), (ii) an implementation of data center network 900 of FIG. 9 a, using ICAS-based multi-unit switching device “ICAS+Multi-unitChassis”, and (iii) an implementation of data center network 900 of FIG.9 a , using ICAS-based stackable switching device “ICAS+StackableChassis” are summarized and compared in Table 5.

TABLE 5 Fat tree + ICAS + ICAS + Multi-unit Multi-unit Stackable ChassisChassis Chassis Intralink (within Pod) N/A 5 5 Interlink (Across Pod) 45 5 Downlink (to Server) 12 15 15 Total 16 25 25 D:U ratio 3 3 3 D:Iratio N/A 3 3 Number of 10 G Interface (for comparison) 96 184.3 184.3QSFP XCVR Module (Watt) 4 4 4 TOR Switch (Watt) 150 200 200 Multi-unitChassis (Watt) 1660 0 0 Spine-side Interlink QSFP XCVR 18432 18800 38000TOR-side Interlink QSFP XCVR 18432 18800 18800 Fabric/TOR-side IntralinkQSFP XCVR 36864 18800 18800 Server-side QSFP XCVR 55296 56400 56400Total QSFP XCVR 129024 112800 (12.6%) 132000 (−2.3%) ASIC in SpineSwitch 2304 1600 1600 ASIC in Fabric Switch 4608 0 0 ASIC in TOR Switch4608 3760 3760 Total Switching ASIC 11520 5360 (53.5%) 5360 (53.5%)Spine Switch (KW) 392.448 327.2 472.0 Fabric Switch (KW) 784.896 0 0 TORSwitch (KW) 986.112 1128.0 1128.0 Total Power Consumption (KW) 2163.4561455.2 (32.7%) 1600 (26.0%) 96 × QSFP Spine Switch (8U) 1536 0 0 96 ×QSFP Fabric Switch (8U) 3072 0 0 48 × QSFP Spine Switch (4U) 0 1600 1600TOR Switch (1U) 4608 3760 3760 ICAS1X5TRIPLE (1U) 0 0 400 ICAS5X21 (2U)0 376 376 Transpose Rack (36U) 0 720 720 ICAS2X9 (1U) 0 0 0 ICAS8X33(4U) 0 0 0 ICAS10X41 (6U) 0 0 0 ICAS16X65 (16U) 0 0 0 Total Rack Unit(U) 9216 6456 (29.95%) 6856 (25.6%) Pod Interlink Bandwidth (Tbps) 7.74.0 4.0 Pod Intralink Bandwidth (Tbps) 7.7 4.0 4.0 Total Data LinkBandwidth (Pbps) 2.2 2.2 2.2 Per Plane Uplink Bandwidth (Tbps) 7.7/plane0 0 Total Spine Uplink Bandwidth (Tbps) 0 150.4 601.6 Total ICAS UplinkBandwidth (Tbps) 0 37.6 37.6 Spine-side Interlink QSFP Cable 18432 1880018800 QSFP Fanout Cable (Transpose Rack) 0 37600 37600 QSFP Fanout Cable(ICAS5X21) 0 19740 19740 TOR-side Interlink QSFP Cable 0 18800 18800TOR-side Intralink QSFP Cable 18432 18800 18800 Spine Switch QSFP Cable0 0 19200 QSFP Fanout Cable (ICAS1X5TRIPLE) 0 0 19200 Total QSFP Cable36864 56400 75600 Total QSFP Fanout Cable 0 57340 76540

As shown in Table 5, the ICAS-based systems require significantly lesspower dissipation, ASICs and space, resulting in reduced material costsand energy.

The above detailed description is provided to illustrate specificembodiments of the present invention and is not intended to be limiting.Numerous modifications and variations within the scope of the presentinvention are possible. The present invention is set forth in theaccompanying claims.

What is claimed is:
 1. A data center network, comprising: a plurality ofnetwork pods, a plurality of first spine planes, and a plurality ofsecond spine planes; wherein each of the plurality of network podscomprises a first ICAS module and a plurality of first layer switches;wherein either the plurality of first spine planes or the plurality ofsecond spine planes is deployed as a data center design option; whereinthe first ICAS module comprises: n port groups, each port groupcomprising n-1 interfaces, wherein n is an integer equal or larger than3; an interconnecting network implementing a full mesh topology, whereineach of the n port groups connects one of the n-1 interfaces to anotherof the n port groups statically, respectively; wherein the n port groupsare indexed with an integer from 0 to n-1; wherein the n-1 interfaces ofthe n port groups are labeled with the same indexes as those ofconnected n port groups; wherein an interface with index j of one of then port groups with index i is connected to an interface with index i ofone of the n port groups with index j, where i is in the range of 0 ton-1, j is in the range 0 to n-1, wherein i does not equal to j, andwherein the interconnecting network comprises all connections betweenthe n port groups; interfaces of each of the plurality of first layerswitches are configured to be grouped into n port groups one ofintralink and interlink respectively; wherein a number of the pluralityof first layer switches is n, and wherein the plurality of first layerswitches is indexed with an integer from 0 to n-1; wherein the pluralityof first spine planes and network pods are interconnected throughinterlinks; wherein interlinks k of the network pods each connects oneof the interlinks of kth of the plurality of first spine plane,respectively, and wherein the interlinks p of the plurality of firstspine planes each connects one of the interlinks of the pth network pod,respectively; a plurality of downlinks to receive and transmit datasignals to and from a plurality of servers; wherein the plurality ofsecond spine planes and the plurality of network pods are interconnectedthrough interlinks; wherein interlinks k of the network pods eachconnects one of the interlinks of the kth second spine plane,respectively, and wherein the interlinks p of the second spine planeseach connects one of the interlinks of the pth network pod,respectively; and wherein a plurality of downlinks to receive andtransmit data signals to and from a plurality of servers.
 2. The datacenter network of claim 1, wherein the network pod further comprises: aplurality of first layer switches whose interfaces are divided intodownlink interfaces, interlink interfaces and intralink interfaces,wherein the downlink interfaces are configured to receive and transmitdata signals to and from a plurality of servers, wherein the interlinkinterfaces of each of the plurality of first layer switches areconfigured into n port group of interlink, wherein the intralinkinterfaces of each of the plurality of first layer switches isconfigured into n port group of intralinks.
 3. The data center networkof claim 1, wherein interfaces of second layer devices of the first ICASmodule are divided into intralink interfaces and uplink interfaces,wherein the intralink interfaces of the first ICAS module are groupedinto the n port groups to connect to the intralink interfaces of thecorresponding n port groups of the plurality of first layer switches,and wherein the uplink interfaces are configured to connect to anexternal network.
 4. The data center network of claim 1, wherein each ofthe plurality of the first spine planes comprises a fanout cabletranspose rack, wherein the fanout cable transpose rack comprises: kgroups of first fiber adapters, each adapter of the k groups of thefirst fiber adapters comprising m interfaces, wherein the k groups ofthe first fiber adapters connect to corresponding ones of k switchesthrough k groups of first fiber cables, wherein the k groups of thefirst fiber adapters also connect to an fiber adapter mounting panel byk groups of first fanout fiber cables, wherein each of a group of ┌p/m┐first fiber adapters connect to a corresponding group of ┌p/m┐ fiberadapters of each switch by ┌p/m┐ first fiber cables, wherein each of thegroup of ┌p/m┐ first fiber adapters connect to the fiber adaptermounting panel by a group of ┌p/m┐ first fanout fiber cables, wherein ┌┐is a ceiling function; and p groups of second fiber adapters, eachadapter of p groups of the second fiber adapters comprising minterfaces, wherein the p groups of the second fiber adapters connect pgroups of second fiber cables to form p groups of interlinks, whereinthe p groups of the second fiber adapters also connect to the fiberadapter mounting panel by p groups of second fanout fiber cables,wherein each of a group of ┌k/m┐ second fiber adapters connects to agroup of ┌k/m┐ second fiber cables to form an interlink, wherein each ofthe groups of the ┌k/m┐ second fiber adapters connects to the fiberadapter mounting panel by a group of ┌k/m┐ second fanout fiber cables,wherein ┌┐ is a ceiling function; wherein the fiber adapter mountingpanel, the k groups of the first fanout fiber cables and the p groups ofthe second fanout fiber cables are cross-connected on the fiber adaptermounting panel, through cross-connection, wherein connections from the kswitches are grouped into p interlinks, each interlink containing oneconnection from each of the k switches, with a total of k connectionsper interlink; and a plurality of third layer switches.
 5. The datacenter network of claim 4, wherein the fanout cable transpose rackconnects to the plurality of third layer switching devices through aplurality of fiber cables; and wherein the fanout cable transpose rackcomprises a plurality of interlinks in the fanout cable transpose rackto connect to a plurality of network pods; wherein connections from theplurality of third layer switching devices are grouped into theplurality of interlinks in the fanout cable transpose rack through thefanout cable transpose rack.
 6. The data center network of claim 5,wherein each of the plurality of interlinks in the fanout cabletranspose rack contains one connection from one of the third layerswitching devices, with each interlink in the fanout cable transposerack having a number of connections, wherein the number equals to anumber of the third layer switching devices.
 7. The data center networkof claim 4, wherein the kth spine plane interconnects the plurality offirst layer switches of each of the network pods through the kthinterlinks working under a full mesh connection has a characteristicsfrom the network with an (n-1, n) bipartite graph, wherein the networkwith (n-1, n) bipartite graph comprises the predetermined number of theplurality of first layer switches and the predetermined number minus onen-1 of the third layer spine switches, and wherein, the plurality offirst layer switches and the third layer spine switches interconnect ina Clos topology.
 8. The data center network of claim 1, wherein each ofthe plurality of second spine planes further comprise a second ICASmodule; wherein the second ICAS module comprises: n port groups, eachport group comprising n-1 interfaces, wherein n is an integer equal orlarger than 3; an interconnecting network implementing a full meshtopology, wherein each of the n port groups connects one of the n-1interfaces to another of the n port groups statically, respectively;wherein the n port groups are indexed with an integer from 0 to n-1; andwherein the n-1 interfaces of the n port groups are labeled with thesame indexes as those of connected n port groups; wherein an interfacewith index j of one of the n port groups with index i is connected to aninterface with index i of one of the n port groups with index j, where iis in the range of 0 to n-1, j is in the range 0 to n-1, wherein i doesnot equal to j, and wherein the interconnecting network comprises allconnections between the n port groups.
 9. The data center network ofclaim 8, wherein the second ICAS modules comprises interfaces dividedinto interlink interfaces and uplink interfaces, wherein the interlinkinterfaces of the second ICAS modules is grouped into a plurality of nport groups, wherein each of the plurality of n port groups of thesecond ICAS modules connects to the corresponding one of the n portgroups of the first layer switches of each of the plurality of thenetwork pods; and wherein the uplink interfaces are configured toconnect to an external network.